Skip to content

Update the redis backend #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Nov 22, 2014
Merged

Conversation

charypar
Copy link
Contributor

Redis backend overhaul to enable real use as storage for bare repositories.

This is my first pull request to libgit2 and I'm not entirely familiar with all the conventions so I may have missed better options to do things. I'll welcome all comments and suggestions of improvement.

What's been done

  • Updated Odb backend to the current libgit2 API
  • Added Refdb backend
  • Changed the key structure to enable multiple repositories in the same redis instance
  • Enabled a static build of the redis backend and linking to a local libgit2

I understand that a redis backend will be slow, especially when used over the network, but I still think it's worth it. My single most important use-case is to enable using git repositories on Heroku or similar app container, which doesn't have a writable file system (not coincidentally, that's also my use-case). It also enables the use of ephemeral, in-memory repositories for cases like undo history without going all out with in-memory storage, etc.

My end goal is to enable switching backends in rugged for ruby/rails apps. I will open a PR to that end soon(-ish) but this can go in first and independently (almost, see below).

For now, the backend is tested through rugged and the tests depend on my hacked-in support for redis backend (see here if interested, but it's probably not worth it). You'll basically have to trust me it works :).

It would be really good to add testing support to libgit2-backends itself. I'm willing to give it a shot if anyone can provide pointers on how best to do it.

Example repo

127.0.0.1:6379> keys rugged:storage/*
 1) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:35bf36aaa1cb0a8e50c2087a18b5cbfc52a92d3"
 2) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:3fc49d757edb6c1add766fbb34f3995810a146c"
 3) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:21e2587d773419407b63992b1425857c7311a99"
 4) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:22001f573a8e7a0768f6116932f0e4456aa529d"
 5) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:refdb:refs/heads/published"
 6) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:d0955f5f1fb29c2dfef1f6957e78458dd9ffd0e"
 7) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:75b7fe1e6981891882f9eb14a123696be4123a8"
 8) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:c8e9c8d242d22db6ed07cac812769d27be20df5"
 9) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:b6dbb5c6c696965e64634cfcb956e84e1a43d2e"
10) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:refdb:HEAD"
11) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:refdb:refs/heads/master"
12) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:fb8d8b4369c084668ab8c62cc50dbc184ff23cc"
13) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:6e457a4e82ede40c4b41dec6df0c90772de8903"
14) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:331a16c1dcb6e2bfeb06aa17aad774c58830a6a"
15) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:5e271d2c8606b8358434ac6fcb461c1c90aa4c2"
16) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:5d95e7bf9ef10fc440fd3e8e687f3173cdf0852"
17) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:5b5b9e43d8eeba5de0ac8e0ec2a26047eaf79fb"
18) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:3d24d025529cb619e8b80c7ce04534c22fbbbc3"
19) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:4efa77fdf5470a5aa7d26a3d6b63738d29470ca"
20) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:f7f9a3c3add8acf7209634c62a777861141cd03"
21) "rugged:storage/b1ff909250a5fda83042abc86f7033f9:odb:cc9096d0d236f765844b75700a72620f959c810"

Issues

I see the following problems with the implementation:

  • No reflog support (because I don't need it)
  • No tests
  • Cmake changes are most likely really bad
  • Built against a slightly older refdb API because of the testing through rugged which isn't entirely up to date with it's vendored libgit2 (this could be a blocker)

/cc @mhodgson, @arthurschreiber

@charypar
Copy link
Contributor Author

charypar commented Mar 4, 2014

I made a few changes to this, namely

  • Fixed reference iteration to only list refs under refs/
  • Added 'AUTH' command support
  • Improved redis connection management to reuse a single connection across the backend instances

@charypar
Copy link
Contributor Author

This is now pretty solid. It's confirmed to work (on Heroku) for our basic usecase. I'd still like to do a couple benchmarks to see how this performs locally and over the network.

But for now, any comments on the implementation itself?

@charypar
Copy link
Contributor Author

@mhodgson @carlosmn any chance for some feedback on this?

Hopefully correct this time:
- iterating with globs assumes the glob is a full name, e.g. 'refs/heads/test-*' or 'H*'
- iterating without a glob uses a default glob of 'refs/*'
This may be wrong, but turning off pkg-config in FindLibgit2.cmake
makes cmake respect library paths given from the command line (with -D)
@charypar
Copy link
Contributor Author

charypar commented Oct 3, 2014

bump.

carlosmn added a commit that referenced this pull request Nov 22, 2014
@carlosmn carlosmn merged commit 45dd8f9 into libgit2:master Nov 22, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants